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Abstract — We consider upper bounds on the error proba- 
bility in channel coding. We derive an improved maximum- 
likelihood union bound, which takes into account events where 
the likelihood of the correct codeword is tied with that of some 
competitors. We compare this bound to various previous results, 
both qualitatively and quantitatively. With respect to maximal 
error probability of linear codes, we observe that when the 
channel is additive, the derivation of bounds, as well as the 
assumptions on the admissible encoder and decoder, simplify 
considerably. 

I. Introduction 

Consider maximum-likelihood decoding, known to be op- 
timal in the sense of average error probability between equi- 
probable messages. What happens when £ false codewords 
share the maximum likelihood score with the transmitted 
one? No matter how such a tie is broken, the average error 
probability given this event is 1 — Computing the 

optimal error probability, taking into account all possible ties, 
is exponentially hard. Can we ignore this event, i.e., assume 
that in case of a tie the decoder is always right or always 
wrong? The answer depends upon both the channel and the 
blocklength. When the likelihood score is a continuous random 
variable, the probability of ties is zero. Also, for long enough 
blocks, the distribution of the score of a word can be closely 
approximated by a continuous one (e.g., using the central-limit 
theorem). However, for small enough alphabet size and short 
enough blocks, the effect of ties on error-probability bounds 
is not negligible. 

We revisit the finite-blocklength achievability results of 
Polyanskiy et al. JT|. For i.i.d. codewords, and when we can 
neglect ties, computation of the exact average error probability 
is not harder than that of the random-coding union (RCU) 
achievability bound. However, ties cannot always be neglected. 
As the RCU bound assumes that ties always lead to errors, 
it can be improved; indeed, we derive a tighter bound. In 
particular, unlike the RCU bound, the new bound is always 
tighter than bounds based upon threshold decoding. 

When it comes to maximal error probability, tie-breaking 
is no longer a mere issue of analysis. Rather, ties have to be 
broken in a manner that is "fair", such that the error probability 
given different messages is balanced. In U, a randomized 
decoder is employed in order to facilitate such fairness. But is 
randomization necessary? We show that at least for additive 
channels (over a finite field), a deterministic decoder suffices. 



II. Notation and Background 

We consider coding over a memoryless channel with some 
finite blocklength n, i.e.: 



V(y\x) = l[V(y i \x i ). 



(1) 



for every x G X n ,y £ y n ■ The channel input and output 
alphabets are arbitrary. For the sake of simplicity, we adopt 
discrete notation; the bounds do not depend on alphabet sizes, 
and the educated reader can easily translate the results to the 
continuous case (which is of limited interest in the context 
of tie-breaking). The codebook is given by Xi, . . . ,Xm, where 
M is the number of codewords (that is, the coding rate is 
R = YnlogM). The decoder produces an estimate m, where 
the transmitted message index is denoted by m. The average 
error probability, assuming equiprobable messages, is given 
by: 



1 M 

— V P(m ^ to|X 



The maximum error probability is given by 



c = max ] 

m=l...M 



'(m ^ m|X = x m ) . 



(2) 



(3) 



For the sake of analyzing the error probability, it is conve- 
nient to consider code ensembles. All ensembles we consider 
in this work fall in the following category. 

Definition 1 (Random conditionally-symmetric ensemble): 
An ensemble is called random conditionally-symmetric 
ensemble (RCSE) if its codewords are drawn such that for 
every different m,j,k G {!,... M} and for every x, x G X n : 



'(X, =i|X r 



' (X fc = x|X m = x) 



(4) 



It is easy to verify, that for an RCSE, all words are identically 
distributed. We can thus define by X a word drawn under the 
ensemble distribution (not necessarily memoryless) P over the 
set X n . Using this input distribution, the information density 
is given by: 



i(x;y) = log- 



(5) 



PV(yY 

where PV(y) is the output distribution induced by P(x) and 



V(y\x). We denote by Y the output corresponding to the 
random input X, and the random variable i(X;Y) is defined 
accordingly. In addition, we define i(X; Y) as the information 
density a codeword X otheiQ than the one that generated Y@ 
The importance of deriving bounds for an RCSE is due to 
the fact that this class includes many interesting ensembles. An 
important special case of RCSE is the pairwise-independent 
ensemble: 

Definition 2 (Pairwise-independent ensemble): A pairwise 
independent ensemble (PIE) is an ensemble such that its code- 
words are pairwise-independent and identically distributed. 
That is, for any two indices i ^ j, 



IX., 



, (X i =x i )=P(x). 



(6) 



We note that the codewords of an RCSE are not necessarily 
pairwise-independent. One example is linear random codes 
with a cyclic generating matrix 0. In this ensemble, a 
codebook is a linear code, such that all the cyclic shifts of the 
any codeword are also codewords. Generally, RCSE (which are 
not necessarily PIE) can be constructed by first drawing a class 
of codewords, and then, randomly (uniformly) drawing the 
codewords from this class. Alternatively, it can be constructed 
by choosing some codeword which defines the class, from 
which all the other codewords will be drawn. 

Finally, the following class of channels turns out to play a 
special role. 

Definition 3 (Additive channels): A channel is additive 
over a finite group Q with an operation, if X = y = Q, 
and the transition distribution V(y\x) is compatible with 

Y = X + N 

where N is statistically independent of X, and "+" denotes 
the operation over f/H 

For example, for modulo-additive channels the alphabet is the 
ring Zq, and addition is modulo q. The importance of additive 
channels stems from the following. 

Lemma 1: Consider an additive channel over Q, and a 
codebook drawn from a PIE with uniform input distribution 
over g n , i.e. P(x) = \Q\~ n Vx e Q n . Then, i(X;Y) is 
statistically independent of (X, Y). 

Proof: For this channel the information density (0 is 
equal to 

Ps(y - x) 



i(x;y) = log 



(7) 



My) ' 

where Pn(-) is the distribution of the noise, and Py(-) is 
the distribution of the channel output. For this channel with 
codebook drawn from a PIE with a uniform distribution over 

'in a random codebook it may happen that the codebook contains some 
identical codewords. Thus it is possible that X = X, as long as they represent 
different messages. 

2 In [T], the notation t(X; Y) is sometimes used; for RCSE, the two are 
equivalent. 

3 The operation "— " over the group, which is uniquely defined by the 
operation "+", such that for any a,b,c € Q : a — b = c iff a = c + b. 



Q n , we have that for every z £ n : 

P(Y-X = z)=P(X + N- X =z) (8a) 

= \g\~ n , (8b) 

since X is uniformly distributed over Q n and statistically inde- 
pendent of (X, N). Therefore, Y— X is statistically independent 
of (X, Y); Moreover, any function of Y — X is also statistically 
independent of (X, Y), in particular Pjy(Y— X) is statistically 
independent of (X, Y). 

Since X is uniformly distributed over Q n , and is statistically 
independent noise, then the channel output Y is also uniformly 
distributed over Q n , i.e. for any y £ Q n : 



My) = \G\~ n , 



(9) 



and hence, Py(Y) is statistically independent of (X, Y). From 
the two observations above, we conclude that £(X;Y) is 
statistically independent with (X, Y). ■ 

III. I.I.D. CODEBOOKS 

Before stating the main results that apply to any RCSE, we 
start by simple bounds that hold for the special case of an i.i.d. 
ensemble. That is, all codewords are mutually independent, 
and each one distributed according to P(X). In this case, 
the average error probability is well known, although hard 
to compute (fl~). Denote: 



W = P(i(X;Y) =i(X;Y)|X,Y) 
Z = ¥(i(X;Y) <i(X;Y)|X,Y) 

Then, for an i.i.d. ensemble |1 Thm. 15]: 

Af-l 



(10a) 
(10b) 



t=0 v 7 



i r Z M-i-e\ 



(11) 



This result stems from the fact that for equiprobable words, 
maximum likelihood (ML) decoding is just maximum infor- 
mation density. We note that I represents the number of com- 
peting codewords that share the maximal information-density 
score with the correct one; given I, the correct codeword will 
be chosen with probability 1 /e+i. If W = (as happens when 
V(Y\x) is a proper density for every x € X), the calculation 
is straightforward. Otherwise, it has exponential complexity. 
Thus, the main burden is with dealing with ties. In order to 
avoid such burden, we suggest the following simple bounds. 

Proposition 1 (Bounds for i.i.d. codebooks): For an i.i.d. 
ensemble, 

1 - Ex,y [{W + Z) M - l \ < e (iid) < 1 - E X ,Y [Z AI - 1 ] . 

(12) 

This result can be shown either from (QT) or directly. For the 
lower bound, in case multiple codewords (including the correct 
one) attain the maximal information density, the correct one is 
always chosen; for the upper bound, it is never chosen under 
such circumstances. Of course, as the upper bound is just the 
first term in (fTTT i. one may tighten it by taking more terms. 
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Fig. 1. The effect of tie-breaking on the performance of i.i.d. codebooks. 
We demonstrate the effect using a BSC with crossover probability 0.3, at 
blocklength n = 100. The triangle- and square- marked solid curves give 
the lower and upper bounds of Proposition [T] respectively. The X -marked 
solid curve is the exact error probability of the i.i.d. ensemble lilt , evaluated 
by taking enough terms in the sum, such that the effect of additional ones 
is numerically insignificant. For reference, the circle-marked dashed curve 
gives the tightest lower bound on the error probability, which holds for any 
codebook [T] Theorem 16]. 



The difference between the lower and upper bounds may be 
quite significant, as demonstrated in Figure Q] 

IV. Bounds for RCSE 

A. Maximum-Likelihood Union Bounds 

When the codewords are not i.i.d., we cannot use anymore 
products of probabilities as done in the previous section. 
However, for providing a lower bound on the error probability, 
we can use a union bound. We derive a result that is close in 
spirit to the RCU bound [1 Theorem 16], which states that 
e (iid) < crcuS where 

£rcu = JEx,y [min {1, (M - 1) • (1 - Z)}} . (13) 

We improve this bound in two ways: First, it is extended to any 
RCSE, and second, the case of equal maximal information- 
density scores is taken into account. 

Theorem 1 (RCU* bound): The average error probability 
of an RCSE satisfies e ( RCSE > < e RCU . where 



£rcu* 



E 



x.v 



1,(M-1)- 1 



(14) 



where the conditional probabilities W and Z are given by (TTOb . 

Proof: Without loss of generality, assume that the trans- 
mitted codeword index is m = 1. The ML decoder will choose 
the codeword with maximal information density; in case of 
equality, it will uniformly draw a winner between the maximal 
ones. Let Cj be the event that the codeword j was chosen in 

4 Indeed, it is noted in [T] Appendix A] that pairwise independence is 
sufficient. 



such a lottery. Denote the following events: 

A 



A i = {i(X i ;Y)>i(X;Y)} 



a 



B j = {i(X j ;Y)=i(X;Y)}. 



(15a) 
(15b) 



A 



Also denote A = Uj=2 A? an< ^ & = Uj= 



B«. Then, the error 



probability is given by: 

e ( RGSE ) = P (A U [B n Ci] \m = 1) (16a) 
= E Xi ,yP(AU [BnCx]\m = l,Xi,Y) . (16b) 
= E Xl ,Ymiri{l,P(AU [BnC 1 }\m= l,Xi,Y)} 

(16c) 



A 



= P(^U [BnCx]\m = l,Xi,Y) 



(16d) 



Using the union bound between events of equality and inequal- 
ity, we have: 



S <F(A\m = l,Xi,Y) 



[Bndlm = l,Xi,Y) . (17) 



Now, the event C\ depends on the the rest of the variables only 
through the number of codewords that achieve equal score. 
Specifically, if there are I impostors, then P(Ci) = 1 /e+i. 
Since the second term is non-zero only if £ > 1, it follows 
that: 

1 



S <F(A\m = l,Xi,Y 
We now use the union bound, as in HI: 



-P(S|m=l,X 1 ,Y). (18) 



E 

J=2 



S < VP(A J -|m = l,X 1 ,Y) 



1 M 

-Y 

3=2 



>(B J |m = l,X 1 ,Y). 

(19) 



Noting that for an RCSE each element in the left (resp. right) 
sum equals 1 — W — Z (resp. W), and substituting this bound 
in d!6db we arrive at the desired result. ■ 
Remark 1: We can give the RCU* bound the following 
interpretation. First, each potential input Xj is given an 
information-density score (equivalent to a likelihood score) 
ij. Then, these scores are fed to a comparison process. The 
process is biased against the correct codeword, in the sense 
that it has to beat each and every impostor. However, each 
pairwise comparison itself is optimal (the correct codeword 
will beat an impostor with lower score), and fair (in case of a 
tie, both codewords are equally likely to win). This comparison 
mechanism is worse than the actual decoder used in the proof, 
since in case the correct codeword shares the maximal score 
with I impostors, it has probability 2~ l to be chosen, rather 
than 1 /e+r, yet, the union bound for both is equal. 



B. Relation to Gallager's Type-I bound 

The following bound is due to Gallager. 

Proposition 2 (Gallager type-I bound fi3§, Sec. 3.3): For 
any constant t: 

e <RCSE) < (20) 



where 

e G .,=P(i(X;Y) < t) + 

+ (M - 1)P (i(X; Y) > t A i(X; Y) > i(X; Y)) . (21) 

Just like the RCU, this bound is based upon a union bound 
for the ML decoder. However, it is inferior to the RCU bound, 
due to the following consideration. Taking the minimum 
between the union and one in the RCU bound is similar to the 
threshold t in (T2T1) . in the way that it avoids over-estimating 
the error probability in cases where the channel behavior was 
"bad". However, the RCU bound uses the optimal threshold 
given X and Y; the Gallager bound uses a global threshold, 
which reflects a tradeoff. Nevertheless, for additive channels 
(recall Definition [3} the local and global thresholds coincide. 

Proposition 3: For any RCSE: 

ec-i > £rcu, (22) 

where eq-i and e R cu are defined in (|2TT > and in ( fT3] l respec- 
tively. If the channel is additive and the code ensemble is PIE 
with uniform distribution over X, then equality holds. 

Proof: For the first part, define the events A = 
(i(X;Y) > i(X; Y)} and T = {i(X; Y) > t} (T c denotes the 
complementary event). Then: 

ercu - Ex,y [min {1, (M - 1) • (1 - Z)}] (23a) 
= P(T C ) -Ex,y [min{l,(M-l) • (1 - Z)}\T C ] 

+ P (T) • E X .Y [min {1, (M - 1) ■ (1 - Z)}\T] (23b) 
< P (T c ) + P (T) • E x ,y [(M - 1)P (A\X, Y)|T] (23c) 
= P(T C ) + (M - l)P(T) -P(A\T) (23d) 
= eg-i (23e) 

For the second part, recall that by Lemma [TJ i(X;Y) is 
statistically independent of (X, Y). Denote by t* the minimal 
threshold t such that 

(M - l)P(i(X;Y) > t) < 1. 

Then (M- 1)P (i(X; Y) > i(X;Y)|i(X; Y) < i*) > 1. Under 
the notation of U from the first part the proof, we have 
that: Ex,y [min {1, f/}|i(X; Y) <t*} = 1, i.e., the inequality 
in (I23cb is equality in this case. ■ 

Remark 2: It follows, that for the BSC, £g-i = £rcu- Indeed, 
it is noted in JT| that for the BSC, the RCU bound is equal 
to Poltyrev's bound 01; this is not surprising, since the latter 
is derived from (l2"TT i (Poltyrev's bound uses linear codes, see 
Section [V] in the sequel). 

Remark 3: Gallager's type I bound can be improved by 
breaking ties, similar to the improvement of RCU*, leading 
to G-I*. An analysis result to Proposition [3] relates G-I* and 
RCU*. 

C. Threshold-Decoding Union Bounds 

The average error probability of an RCSE can be further 
lower-bounded using the sub-optimal threshold decoder 
This decoder looks for a codeword that has a likelihood score 
above some predetermined threshold. In JT] Theorem 18] a 



union bound is derived for such a decoder, where if multiple 
codewords pass the threshold, the winner is chosen uniformly 
from among them|f| The resulting "dependence testing" (DT) 
bound is given by: 

M — 1 

e DT 4 P (i(X; Y) < 7 ) + — — P (i(X, Y) > 7 ) , (24a) 



where the optimal threshold is given b}0 

Af-1 



7 



lOE 



(24b) 



A troubling behavior, demonstrated in [ l j using the binary 
erasure channel (BEC), is that sometimes 6rcu > £dt- This is 
counter-intuitive since the DT bound is derived from a sub- 
optimal decoder. We find that this artifact stems from the fact 
that the RCU bound ignores ties, and prove that the improved 
bound, denoted by RCU*, always satisfies £rcu* < £dt- To 
that end, we prove a (very slightly) improved bound for the 
threshold decoder, that is closer in form to the ML bounds ([Oi l 
and (fZTJi. It uses the following definitions (cf. (fTUli). 



W q =P(g(i(X;Y)) = g(i(X;Y))|X,Y) 
Z q =F (q(i(X;Y)) < q(i(X;Y))\X,Y) 

where q(i) is the indicator function: 

q(i) = 1{*> 7 }- 
Proposition 4: For an RCSE, 



(RCSE) 



where 

L 

Ctu - 



^X,Y 



mnJ 1, (M — 1) • ( 1 



(25a) 
(25b) 

(25c) 

(26) 

(27) 



Furthermore, exu < £dt- 

Proof: For proving achievability, consider a decoder iden- 
tical to the ML decoder, except that before comparing the 
words, the information-density scores are quantized according 
to d25cl >. For the comparison to the DT bound, 

M-l. 



X,Y 



{i(X;Y)< T } 



i(X,Y) > 7 |X,Y) 

(28a) 



> E- 



> 3 



X,Y 
ip 

2 

X,Y 



min |l, (M - 1) 
»(X,Y) > 7 |X,Y) 

. r m — i 

mm < 1, • 



{ l (X;Y)< 7 } 



[l{i(X;Y)< 7 } 



+P (i(X,Y)> 7 |X,Y)] 

£tu- 



(28b) 

(28c) 
(28d) 



5 In fact, the proof states that the "first" codeword to pass the threshold is 
selected. However, such ordering of the codewords is not required. 

6 In (§], the threshold is further optimized, depending on the competing 
codeword and on the received word 



Remark 4: It is not obvious that the optimal threshold for 
the TU bound is 7 of ( 124bl i. However, it is good enough for 
our purposes. 

Proposition 5: For any channel, crcu* < £tu- Thus, the 
RCU* bound is tighter than the DT bound, i.e.: 

£rcu* < £dt- 

Proof: Recalling Remark [TJ the RCU* bound reflects 
optimal (ML) pairwise decision. Thus, necessarily the pairwise 
error probabilities satisfy: 



W 



> Z n 



Wn 



(29) 



Remark 5: In fact, the case of the BEC, where £rcu* = 
e-ru = min(l,eDT) is ver y special. In the BEC, an impostor 
cannot have a higher score than the true codeword; if the chan- 
nel realization is such that the non-erased elements of x and 
x are equal, then i(x;y) = i(x;y), otherwise i(x;y) = —00. 
Thus, 



£rcu* — E x ,y 



min < 1 



(M - 1)W 



(30) 



Let k be the number of non-erased symbols out of the block 
of n, then W = 2~ fe . Consequently, (M - l)W/2 > 1 if and 
only if i(x,y) < 7, where 7 is given by ( 124bt . 

D. Performance Comparison 

Comparison of the different union bounds is given in 
Figure [2] In particular, the effect of tie-breaking on the bounds 
is shown by the comparison of the RCU bound (Tl3T > and the 
RCU* bound (fT4l . Notice that this bound depends on the 
ensemble. Due to Lemma [TJ the computation of the RCU 
and RCU* bounds for PIE becomes simple, hence show the 
bounds for this ensemble. Since an i.i.d. ensemble is also PIE, 
the exact error probability for i.i.d. ensemble (fiH is given as 
a reference. 

V. Linear Codes 

Most known good codes are linear. Beyond that, linear 
codes have an important role for two reasons. First, they allow 
to improve performance (both capacity and error probability) 
in many network problems (see, e.g., Q, JU). Second, for 
some channels, the average error probability and maximal 
error probability coincide for linear codes. 

A. The Dithered Linear Codes Ensemble 

For any finite field ¥ q of cardinality q, we define the dithered 
linear codes ensemble by 



X, = Hi,, 



D. 



(31) 



Here, all operations are defined over the field, all elements of 
the n x k generator matrix H and length- k dither vector D are 
drawn uniformly and independently from the field elements, 
and {\Vj } are all fc-vectors over the field. It follows that the 
codebook size is M = q k . An important special case is when r 
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Fig. 2. The effect of tie-breaking on the performance of PIE codebooks of the 
different union-bounds. We demonstrate the effect using a BSC with crossover 
probability 0.3, at blocklength n = 100. The triangle-marked dashed curve is 
the DT bound (24) . The asterisk-marked dashed curve is the TU bound 427t . 
The dashed curve is the RCU bound H3\ . The diamond-marked solid curve is 
the RCU* bound j 14} . For reference, we repeat two of the curves of Figure[T] 
The X -marked solid curve is the exact performance of the i.i.d. ensemble jilt , 
while the circle-marked dashed curve is the lower bound for any codebook [T] 
Theorem 16]. The non-smoothness of some of the curves is not an artifact, 
but comes from the fact that they involve integers. 



is prime, and modulo arithmetic is used, e.g., binary (dithered) 
linear codes. 

By 13, any dithered linear codes ensemble over this field is 
PIE. Consequently, the RCU* bound applies to this ensemble. 
Further, it is proven in JTJ Appendix A] that for a class 
of channels, which includes the BSC and the BEC, there 
exists a randomized ML decoder such that the maximal error 
probability e coincides with the average one. 

B. Additive Channels 

We now restrict our attention to channels that are additive, 
in the sense of Definition [3] Further, assume that the channels 
are additive over a finite field, which is the same field over 
which the code is linear. Clearly, in this situation the dither 
does not change the distance profile of the codebook, thus it 
suffices to consider the linear codes ensemble 



(32) 



where again H is i.i.d. uniform over ¥ q . More importantly, in 
order to achieve good maximal error probability, there is no 
need to use randomized decoders. 

Theorem 2: For any channel that is additive over a finite 
field, for an ensemble of linear codes over the field, there 
exists a deterministic decoder satisfying: 

e < £rcu* 

Remark 6: Recall that the size of linear code is M = q k 
for an integer k. Thus, the theorem does not give e for all 
n,M. 



Proof: Let Ox, ... , fljif be a partition of the output space 
y n into decision regions (for any 1 < m < M, fi m is 
associated with codeword m). A partition is optimal in the 
average error probability sense, if and only if it satisfies: 

ftm C{ye y|Vm' + m ■ V{y\x m ) > V(y\x m ,)} (33a) 
O m 3{ye y n \Vm' ^ m : V(y\x m ) > V(y\x m ,)} , (33b) 

and for all m ^ m! f2 m n Vt m ' = 0- By 0, we have that for 
an additive channel, d33l is equivalent to 

O m C{ye F£|Vm' ^ m : P N (y - x m ) > P N (y - x m -)} . 

(34a) 

m 3{ye F r 9 l |Vm' ^ m : P N (y - x m ) > F N (y - x m ,)} • 

(34b) 

Since for any such optimal partition e < 6rcu* , it is sufficient 
to show that there exists a partition satisfying (l34t for which 

6 = 6. 

Let C C F™ be a linear code, and without loss of generality 
assume that Xi = 0. We construct decoding regions for C in 
the following way. Recall that C induces a (unique) partition 
of the linear space F™ into disjoint cosets Si, ... , <S n -fc, where 
each coset is a translation of the linear code. For each coset 
Sj let the coset leader 1 be some word that may belong to 
fii according to d34at . By definition, all coset elements of the 
coset can be uniquely labeled as 

y j.m = y j,l > X m- 

Assign each word y^ m to Vt m . It then satisfies that for all 
m' = l, ...,M 

Ps(yj,m - x m) = Ps(yj,i) (35a) 
> Ps(y jtl - (x m / - Xm)) (35b) 

= Ps(yj,m -*m'), (35c) 

where in d35bb we have used the facts that y 3 1 satisfies d34ab 
and that the sum of codewords is a codeword. It follows, that 
the partition indeed satisfies (l34l . To see that e = e, we have 
that for all m = 1, . . . , M: 

P (Y G n m \X = Xm) = P (X + N e O m |X = x m ) (36a) 

= P(NeO m -x m |X = x m ) (36b) 

= P(NeOi|X = x m ) (36c) 

= P(Nefii) (36d) 

= P(xi+NgOi) (36e) 

= P(Yef2i|X = xi), (36f) 

where ( I36cb is due to the construction of f2 m , and d36db is 
since the noise is statistically independent of the channel input. 
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Remark 7: The partition used in the proof is not unique, in 
the sense that for some cosets the choice of the coset leader is 
arbitrary. However, for any such choice the coset is partitioned 
in a fair way between the decision regions. 



